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5 [0001] A portion of the disclosure of this patent document contains material that is subject 
to copyright protection. The copyright owner has no objection to the xerographic 
reproduction by anyone of the patent document or the patent disclosure in exactly the form it 
appears in the U.S. Patent and Trademark Office patent file or records, but otherwise reserves 
all copyright rights whatsoever. 
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CROSS-REFERENCES TO RELATED APPLICATIONS 
[0002] The entire contents of the following applications are incorporated herein by 
reference for all purposes: 

[0003] (1) U.S. Application No. 09/728,560, filed November 30, 2000 (Attorney Docket 
15 No. 1 5358-0062 10US); 

[0004] (2) U.S. Application No. 09/728,453, filed November 30, 2000 (Attorney Docket 
No. 15358-006220US); and 

[0005] (3) U.S. Application No. 09/52 1 ,252, filed March 8, 2000 (Attorney Docket No. 
15358-006300US). 
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BACKGROUND OF THE INVENTION 
[0006] The present invention relates to a note-taking system, and more particularly to 
techniques that improve a user's note-taking experience in an environment where information 
of various types is being recorded. 

25 [0007] Several techniques have been developed in the past few years for capturing 

information during a meeting, lecture, or other like presentation. For example, during a 
presentation, slides (e.g., from a PowerPoint presentation) and video clips may be shown on a 
projection device while the presenter discusses information related to the slides and/or video 
clips. Attendees of the meeting may also join in the discussion. The presenter may also write 
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on a whiteboard during the presentation. In such an environment, one or more information 
capture devices may be provided to record the multimedia information output during the 
presentation. For example, a video camera may be used to record the slides/video clips 
presented on the projection device or the actual scene of the presentation. Audio devices 
5 such as microphones may be used to capture the audio information spoken by the presenter or 
the attendees of the meeting. A whiteboard capture device may be used to capture 
information written on the whiteboard. Various other types of devices may be used to record 
other types of information output during the presentation. 

[0008] The recorded information captured by the various capture devices is usually stored 
1 0 and can be accessed or consumed by a user after the presentation. For example, a user who 
could not attend the meeting can access the recorded information and play back the 
information after the meeting. In this manner, the archived information is available for use 
subsequent to the meeting. 

[0009] During a presentation attendees of the presentation usually capture information 
1 5 presented during the presentation by taking notes. Although the traditional technique of 

taking notes has been to use a paper and pencil/pen, an increasing number of users now use 
computing devices such as laptops, personal digital assistants (PDAs), tablet PCs, etc. to take 
notes during presentations. However, even with these devices, it is usually quite difficult for 
the note-taker to capture in his/her notes information of various types (e.g., audio 
20 information, slides, video information, etc.) that is output during the presentation. Although, 
the information during the presentation is being recorded, the recorded information is not 
available for use by the note-taker until after the presentation has completed. Accordingly, 
the recorded information cannot be used by the note-taker while the presentation is in 
progress. 

25 [0010] Accordingly, there is a need for techniques that improve a user's note-taking 
experience. 

BRIEF SUMMARY OF THE INVENTION 
[0011] Embodiments of the present invention improve a user's note-taking experience. 
30 While taking notes for a presentation, a user can request one or more portions of information 
recorded during the presentation to be inserted into the user's notes document. User 
information requests that cannot be immediately processed are stored or cached in the notes 
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document for later processing. These cached requests are processed and the requested 
information inserted into the user's notes document at a later time. 

[0012] According to an embodiment of the present invention, techniques are provided for 
taking notes in an environment comprising a server configured to respond to requests 
5 requesting a portion of stored information, the stored information comprising first 

information captured by one or more capture devices during a first presentation, and a note- 
taking device for taking notes during the first presentation in a notes document. A first 
request is received at the note-taking device during the first presentation to insert a first 
portion of the first information in a first location in the notes document. A determination is 
10 made if the first request can be processed. The first request is stored in the notes document 
upon determining that the first request cannot be processed. 

[0013] According to an embodiment of the present invention, a determination is made, 
subsequent to storing the first request in the notes document, if the note-taking device can 
communicate with the server. The first request is processed upon determining that the note- 
1 5 taking device can communicate with the server. According to an embodiment, the first 
portion of the first information is inserted in the notes document in the first location. 

[0014] The foregoing, together with other features, embodiments, and advantages of the 
present invention, will become more apparent when referring to the following specification, 
claims, and accompanying drawings. 

20 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0015] Fig. 1 is a simplified block diagram of a system that may incorporate an 
embodiment of the present invention; 

[0016] Fig. 2 is a simplified high-level flowchart depicting a method of processing requests 
25 for inserting recorded information in a user's notes document according to an embodiment of 
the present invention; 

[0017] Fig. 3 depicts a list of templates including a template that provides note-taking 
functionality according to an embodiment of the present invention; 

[0018] Fig. 4 depicts a section of a Microsoft Word™ tool bar showing a menu bar 
30 comprising user-selectable options for inserting portions of recorded information into a notes 
document according to an embodiment of the present invention; 
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[0019] Fig. 5 depicts a user interface that displays information regarding sessions that the 
user is authorized to access information from according to an embodiment of the present 
invention; 

[0020] Fig. 6 displays a user interface that allows a user to browse through previously 
5 recorded session information when the information is accessible to the user's note-taking 
device according to an embodiment of the present invention; 

[0021] Fig. 7 A depicts a Microsoft Word™ interface 700 that has been configured to 
provide options for taking notes and that display a notes document with a cached request 
according to an embodiment of the present invention; 

10 [0022] Fig. 7B depicts the notes document of Fig. 7A wherein the cached information 
request has been processed according to an embodiment of the present invention; and 

[0023] Fig. 8 is a simplified block diagram of a note-taking device 108 that may be used to 
perform processing according to an embodiment of the present invention. 

1 5 DETAILED DESCRIPTION OF THE INVENTION 

[0024] In the following description, for the purposes of explanation, specific details are set 
forth in order to provide a thorough understanding of the invention. However, it will be 
apparent that the invention may be practiced without these specific details. 

[0025] Fig. 1 is a simplified block diagram of a system 100 that may incorporate an 
20 embodiment of the present invention. System 100 depicted in Fig. 1 is merely illustrative of 
an embodiment incorporating the present invention and does not limit the scope of the 
invention as recited in the claims. One of ordinary skill in the art would recognize other 
variations, modifications, and alternatives. 

[0026] As depicted in Fig. 1, system 100 includes one or more information capture devices 
25 102 that sire configured to capture (or record) information during a presentation. For 

purposes of this application, a "presentation" is intended to refer to any event or period of 
time during which information is presented or output. Examples of presentations include 
meetings, lectures, discussions, classes, conferences, speeches, demonstrations, playback of 
previously recorded information, and the like. A presentation need not be contiguous in time 
30 and can be spread out over non-contiguous time periods. For example, a presentation may be 
spread out over several days and may be interspersed by other events. 
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[0027] For purposes of this application, an "information capture device" is intended to refer 
to any device, system, or apparatus that is configured to capture or record information of one 
or more types. Various different types or streams of information such as audio information, 
video information, slides or images, whiteboard information, text information (e.g., user 
5 contributed comments), etc. may be output during a presentation and captured by information 
capture devices 102. Examples of information capture devices 102 include microphones, 
video cameras, analog or digital photo cameras, scanners, screen capture devices (e.g., a 
device configured to capture information printed on a whiteboard), presentation recorders, 
etc. Information capture devices 102 may also be configured to capture temporal information 
10 when the information is captured. 

[0028] A presentation recorder is a device that is able to capture information presented 
during a presentation, for example, by tapping into and capturing streams of information from 
an information source such as a computer executing a PowerPoint application and displaying 
slides from a *.ppt file. A presentation recorder may be configured to tap into the video 

1 5 output of the computer and capture keyframes every time a significant difference is detected 
between displayed video keyframes of the slides. The presentation recorder may also be 
configured to capture other types of information such as audio information, video 
information, slides information stream, etc. Time stamps indicating time of capture may also 
be stored by the presentation recorder. The temporal information indicating when the 

20 information was output or captured may be used to synchronize the different types of 

captured information streams. Examples of presentation recorders include a screen capture 
software application that allows recording of slides and time elapsed for each slide during a 
presentation. Examples of presentation recorders are described in U.S. Application No. 
09/728,560, filed November 30, 2000 (Attorney Docket No. 15358-006210US), U.S. 

25 Application No. 09/728,453, filed November 30, 2000 (Attorney Docket No. 15358- 

006220US), and U.S. Application No. 09/521,252, filed March 8, 2000 (Attorney Docket No. 
15358-006300US), whose contents are incorporated herein by reference for all purposes. 

[0029] In the embodiment depicted in Fig. 1, the information recorded by information 
capture devices 102 is pushed to server 104 ("presentation server 104") that caches and 
30 maintains the recorded information. The information may be pushed to presentation server 
104 in a continuous stream or in bursts. The recorded information may include streams of 
information of various types including slides, images, audio segments, (e.g., MP3 audio 
segments), video clips or segments, whiteboard information segments, text information, etc. 
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[0030] Presentation server 104 may store the information received from information 
capture devices 102 in a memory location accessible (locally or via a network) to presentation 
server 104. In one embodiment, presentation server 104 is configured to coordinate access to 
the stored information. For example, presentation server 104 is configured to respond to 
5 requests requesting portions of the recorded information. 

[0031] According to an embodiment of the present invention, the recorded information may 
be organized into "sessions' 1 106. For purposes of this application, a "session" is a collection 
of recorded information. For example, a session may correspond to information streams 
captured by information capture devices 1 02 during a certain time period. For example, 

10 information recorded during a first presentation (e.g., a meeting between 1 :00pm and 
2:00pm) may be stored as a first session while information recorded during a second 
presentation (e.g., a meeting between 3:00pm and 5:00pm) may be stored as another session, 
A session may also store information recorded over different time periods or different 
presentations. For example, information captured during a series of lectures may be stored as 

15 a single session. Presentation server 104 is configured to control access to the various 
sessions information. 

[0032] In the embodiment depicted in Fig. 1, a user uses a computing device 108 ("note- 
taking device 108") to take notes. The notes may include text, figures, etc. entered by the 
user. According to an embodiment of the present invention, the notes may also include 

20 portions of recorded information (e.g., multimedia streams or objects). Examples of note- 
taking device 108 include a personal computer, a portable or laptop computer, a personal 
digital assistant (PDA), a tablet PC, etc. The user or note-taker may use an application 
("composing application 1 10") 110 executing on note-taking device 108 to take notes in a 
document 112 ("notes document") created by composing application 110. Examples of 

25 composing applications 110 include applications such as Microsoft Word™, Microsoft 

Excel™, WordPerfect™, text editors (e.g., TextEdit, WinEdit), and the like. Any application 
that can be used to create or open a document and write information to the document can be 
classified as a composing application. For example, the user may open a document 112 using 
Microsoft Word™ executing on note-taking device 108 and take notes in the document. The 

30 notes document 112 may be stored on note-taking device 108. 

[0033] According to an embodiment of the present invention, while taking notes during a 
presentation, a user can specify that portions of information recorded by capture devices 102 
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during the presentation be incorporated or inserted into the user's notes. The portion may 
correspond to a slide or other image, an audio segment, a video segment, a text segment (e.g., 
user contributed text notes), etc. A portion may correspond to a multimedia object included 
in the information captured by information capture devices 102. For example, the user may 
5 request that: a slide being presently shown on a projection device be inserted at a particular 
location in the user's notes; an audio segment comprising the last 5-minutes of recorded audio 
information be inserted into the user's notes; a video clip being presently shown be inserted 
into the user's notes; and the like. Embodiments of the present invention thus enable a user to 
incorporate information recorded during a presentation or portions thereof into the user's 
10 notes while the presentation is proceeding. Recorded information of various types that is 

captured during the presentation may be incorporated into the user's notes. Details related to 
how a user can incorporate such information in the user's notes are provided below. 

[0034] Whether or not a user's request to insert information in the notes can be processed 
generally depends upon whether note-taking device 108 can communicate with presentation 

15 server 104 and receive the requested information for presentation server 104. Note-taking 
device 108 may or may not be communicatively coupled (i.e., can communicate with 
presentation server 104) to presentation server 104. This on/off nature of the connection 
between note-taking device 108 and presentation server 104 is indicated by dotted line 114 
depicted in Fig. 1. For example, note-taking device 108 may be connected or may not be 

20 connected to presentation server 104 during (or after) a presentation when the user is taking 
notes using note-taking device 108. The connection between the note-taking device 108 and 
the presentation server 104 may be volatile in some situations where it comes and goes (e.g., 
a weak wireless connection) during a presentation. 

[0035] When note-taking device 108 is connected to presentation server 104, then note- 
25 taking device 108 can access information recorded during the presentation by information 

capture devices 102 and pushed to presentation server 104. Note-taking device 108 may not 

have access to the captured information when not connected to presentation server 104. 

According to the teachings of the present invention, the ability of a user to take notes, 

including specifying portions of the captured information to be inserted in the notes, is not 
30 dependent on whether or not note-taking device 108 is communicatively coupled to 

presentation server 104 (i.e., not dependent on whether note-taking device 108 has access to 

the captured information or not). 
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[0036] According to an embodiment of the present invention, when note-taking device 108 
can communicate with presentation server 104, user requests for portions of recorded 
information to be incorporated into the user's notes are communicated from note-taking 
device 108 to presentation server 104. Presentation server 104 responds to the requests by 
5 sending the request information portions to note- taking device 108. The requested 

information portions received from the presentation server 104 are then incorporated into the 
user's notes in locations identified by the user. In this manner, portions of the information 
being recorded during the presentation can be inserted into the user's notes during the 
presentation. 

10 [0037] According to an embodiment of the present invention, when note-taking device 108 
cannot communicate with presentation server 104, user requests for portions of recorded 
information to be incorporated into the user's notes are cached (or stored) in the notes 
document that the user is composing. Information identifying locations within the notes 
document where the requested information portions are to be placed may also be cached in 

1 5 the notes document. A visible place holder or marker may be placed in the notes document at 
a location where the requested information is to be inserted. The place holder or marker may 
indicate contextual information (e.g., a slide number from a particular session) regarding the 
requested information. Accordingly, a user need not have access to the recorded information 
when the information request is made. 

20 [0038] Information requests that are cached in the notes document are then processed at a 
later time. For example, the cached requests may be automatically processed when note- 
taking device 108 detects that communication with presentation server 104 has been 
established. Note-taking device 108 may periodically check if communication with the 
presentation server 1 04 has been established. The check to determine if communication has 

25 been established may be performed upon receiving a signal generated in response to a user 
action (e.g., user opens the notes document, user explicitly requests a communication check) 
or in response to an automated signal (e.g., a signal generated by a timer, a signal generated 
in response to some event related to the notes document). 

[0039] As described above, the information recorded by information capture devices 102 
30 and pushed to presentation server 104 may be stored and organized into sessions. According 
to an embodiment of the present invention, a user's access to the recorded information is 
regulated on a per session basis. In this embodiment, a user or note-taker is allowed to 



request information stored in a session only if the user is authorized to access information 
from that session. As a result, in this embodiment, a user has to secure authorization for a 
session before the user request recorded information from the session. 

[0040] In the embodiment depicted in Fig. 1, authorization device 116, presentation server 
5 104, and an authorization module 118 executing on note-taking device 108 perform the 

authorization operations. In this embodiment, a user gains authorization by logging in using 
authorization module 118 and authorization device 116. Authorization module 1 18 is 
independent of composing application 110. 

[0041] According to an embodiment of the present invention, the authorization process is 
10 initiated when authorization module 118 communicates login information from the user's 
note-taking device 108 to authorization device 116 which in turn communicates the 
information to presentation server 104. The login information may include username 
information, IP address of note-taking device 108, clock information of the note-taking 
device 108, and other information used for authorization. 

15 [0042] Presentation server 104 responds by communicating authorization information to 
authorization device 116 which in turn communicates the information to note- taking device 
108. According to an embodiment of the present invention, the authorization information 
comprises a session identifier that identifies the session for which authorization is granted for 
the user. For example, if the user is attending a presentation, the session identifier may 

20 identify a session storing information recorded for the presentation that the user is attending. 
Presentation server 104 may also communicate other information as part of the authorization 
information such as a login key, the presentation server's host IP address that the note-taking 
device 108 can use to communicate with the presentation server 104, and other information. 
Presentation server 104 can be configured to implement several strategies of authorization 

25 depending on the access level of a session in a manner similar to those of a file system 
protection. According to an embodiment of the present invention, if a session is declared 
private, only password protected users are authorized to access session information. These 
password protected user accounts may be created in advance through another process such as 
conference registration and made known to the user. Alternatively, a login account may be 

30 dynamically created if the authorization request is received through an infrared authorization 
device during the presentation. If a session is declared public, then any authorization request 
is granted. 
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[0043] A user may be also gain authorization to multiple sessions. For example, in 
addition to the current session storing recorded information for the current presentation that 
the user is attending, the user may have previously attended three other presentations and 
gotten authorization to sessions storing recorded information for the three presentations. 
5 Different session identifiers may be used to identify different sessions. Presentation server 
104 may store a list of authorized session identifiers for each user. The status of the 
login/authorization process may also be output to the user using note-taking device 108. 

[0044] According to an embodiment of the present invention, infrared (ER) technology may 
be used to accomplish the authorization process. For example, authorization device 116 may 

10 be an IR receiver/transmitter and authorization module 118 may be an IR module configured 
to communicate with an IR receiver. In one such embodiment, ER module 118 may be 
configured to automatically start when note-taking device 108 is started (e.g., when a laptop 
boots up). ER Module 1 18 is configured to monitor the ER port on note-taking device 108 and 
send protocol information to an ER authorization device 116 when it comes within range of 

15 authorization device. For example, in one embodiment, when a note-taking device 108 

comprising an ER module 118 comes within range of an ER authorization device 1 16, it sends 
the username information, note-taking device's IP address, and clock information to 
authorization device 116. In return, note-taking device 108 receives a session identifier and 
other information (e.g., a login key, presentation server's IP address) from authorization 

20 device 116. In one embodiment, the IR authorization module 1 18 is implemented using 
VC++ (Visual C++) and ER communication is based on ER sockets. 

[0045] As stated above, the status of the login/authorization process may also be output to 
the user using note-taking device 108. For example, in the case where authorization module 
1 18 is automatically initiated by detecting an ER authorization device 1 16 in alignment, a 
25 message may be displayed on the note-taking device 108 indicating success or failure of the 
login/authorization process. An audible signal (e.g., a bell sound) may also be used to 
indicate the status of the login and authorization. 

[0046] Using ER technology, the physical presence/location of the user's note-taking device 
108 may initiate the authorization and login process. For example, one or more ER 
30 authorization devices 116 may be placed at convenient locations in a conference room. 
When the conference room is used for a presentation, attendees of the presentation may 
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automatically login and gain access to information recorded for the presentation by bringing 
their note-taking devices within range of the IR authorization devices 116. 

[0047] As another example, in an environment where a table is used for meetings, IR 
authorization devices 116 may be placed on the table. When people attending a meeting sit 
5 down around the table and start their note-taking devices 108, they are automatically logged 
in when the IR authorization modules 118 executing on their note-taking devices 108 come 
within range of the IR authorization devices 1 16. In this manner, an "intelligent 11 
environment may be created that facilitates recording of information and note-taking 
according to the teachings of the present invention. 

10 [0048] Various other technologies besides IR technology may also be used to accomplish 
the login and authorization process. For example, a web page may be provided for the user to 
login and gain authorization. 

[0049] Fig. 2 is a simplified high-level flowchart 200 depicting a method of processing 
requests for inserting recorded information in a user's notes document according to an 

1 5 embodiment of the present invention. The method depicted in Fig. 2 may be performed by 
software modules executed by note-taking device 108, hardware modules of note-taking 
device 108, or combinations thereof. Flowchart 200 depicted in Fig. 2 is merely illustrative 
of an embodiment of the present invention and is not intended to limit the scope of the 
present invention. Other variations, modifications, and alternatives are also within the scope 

20 of the present invention. 

[0050] As depicted in Fig. 2, processing is initiated in step 202 upon receiving a request 
during a presentation to insert into the user's notes a portion of information recorded and 
stored in a session. The request may identify the type of information requested, for example, 
a slide or other image, an audio segment, a video segment, a text segment, etc. Multiple 

25 types of information (e.g., a slide along with an audio clip) may be requested in a single 
request. The request may also identify a particular location within the note-takers' notes 
document where the requested information is to be inserted. The request may include 
information specifying a session identifier identifying a session from which information is 
requested. The session identifier may identify a session storing information for the 

30 presentation in progress or any other session storing information recorded from a previous 
presentation to which the user has gained authorization. According to an embodiment of the 
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present invention, each information request received in step 202 is assigned a unique request 
identifier. 

[0051] The information request in step 202 may be received at any time whether or not 
note-taking device 108 is communicatively coupled to presentation server 104. Accordingly, 
5 the user can request the information while taking notes irrespective of whether the requested 
information is accessible to note-taking device 108. 

[0052] A note-taker may use various techniques to configure a request. In one 
embodiment, a user interface may be displayed on note- taking device 108 that allows the user 
to specify information to be incorporated into the user's notes. The user interface for 

10 specifying the information to be inserted may be integrated with the user interface provided 
by composing application 110 that is used to take the notes. For example, the user interface 
of a composing application 110 such as Microsoft Word™ may be customized to provide 
options (e.g., buttons, menus, etc.) that allow a user to configure requests requesting a portion 
of recorded information to be inserted into the user's notes. One such embodiment is 

1 5 described below in further detail. 

[0053] According to one technique, the information to be inserted into the user's notes may 
be based upon the time when the user makes the request and the type of information selected 
by the user. In this embodiment, user-selectable options (e.g., buttons, menus, etc.) may be 
provided which allow the user to select the type (e.g., slide, audio segment) of recorded 
20 information to be incorporated into the user's notes. In this embodiment, the selected 

information type along with the time when the selection is made and the session identifier 
form the information request. According to an embodiment of the present invention, the 
location of the cursor within the notes document when the option is selected indicates the 
location in the notes document where the requested information is to be placed. 

25 [0054] For example, a plurality of buttons may be displayed each corresponding to an 
information type such as slides, audio, video, text, etc. If the user wants to insert the 
presently displayed slide in the user's notes, the user can move the cursor to a location in the 
notes document where the slide is to be inserted and then select the "slides" button. This 
generates an information request requesting that the currently displayed slide (i.e., the slide 

30 being displayed when the user selected the "slides" button) be inserted in the cursor location 
in the user's notes. The user may select other buttons corresponding to the other types of 
information to configure requests to insert portions of those particular types of information. 
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For example, if the user selects the "audio 11 button, then a request for audio information is 
generated. The amount of audio information requested may be based upon the time when the 
request was made. In one embodiment, a predetermined and user configurable time buffer 
relative to the time of selection (e.g., 1 minute prior to the time of selection) may be used to 
5 identify the amount of audio information to be incorporated. In other embodiments, the user 
may be allowed to specify the time length of the audio information to be inserted in the user's 
notes. 

[0055] According to another technique, a user may specifically identify the information to 
be incorporated into the user's notes. For example, the user may specify a slide number 
10 identifying a slide (which may have been previously displayed during the presentation) to be 
inserted into the user's notes. Likewise, the user may specify a portion of other types of 
recorded information (e.g., video clip shown during the first 5 minutes of the presentation, 
audio information spoken by a particular speaker during the presentation) to incorporated into 
the user's notes. 

1 5 [0056] As described earlier, in some embodiments of the present invention, a user may be 
authorized to access information from multiple sessions. For example, in addition to the 
presently recorded session, the user may also be authorized to access information from other 
previously recorded sessions. In such embodiments, the user can configure requests 
requesting information not only from a session corresponding to the present presentation but 

20 also from other sessions. For each request, the user may specify the session (e.g., using a 
session identifier) from which information is requested. 

[0057] In step 204, it is determined if the information request received in step 202 can be 
processed. A request can be processed if the requested information is accessible to note- 
taking device 108. The requested information may be accessible via presentation server 104 
25 or in some situations may be accessible locally (i.e., if the requested information is locally 
cached on note-taking device 108). A request cannot be processed if the requested 
information is not accessible. 

[0058] Accordingly, in most situations, whether or not a user's request to insert information 
in the notes can be processed generally depends upon whether note-taking device 108 can 
30 communicate with presentation server 104 and receive the requested information from 
presentation server 104. In one embodiment, the request can be processed if note-taking 
device 108 can communicate with presentation server 104. Accordingly, a check is made in 
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step 204 to see if the note-taking device 108 and presentation server 104 can communicate 
with each other. If it is determined that the note-taking device 108 and presentation server 
104 can communicate with each other, then it is deemed that the request can be processed and 
processing continues with step 206. If it is determined that the note-taking device 108 and 
5 presentation server 104 cannot communicate with each, then it is deemed that the request 
cannot be processed and processing continues with step 214. 

[0059] In certain situations, it may be determined in step 204 that the information request 
received in step 202 cannot be processed even though the note-taking device 108 and 
presentation server 104 can communicate with each other. For example, the information 
10 requested may not yet be available. In this situation processing continues with step 214. 

[0060] In step 206, an information query is composed based upon the information request. 
According to an embodiment of the present invention, the information query comprises a 
session identifier identifying the session from which information is requested, a login key 
received during authorization, and information identifying the requested information. The 
1 5 query composed in step 206 is then communicated to the presentation server 104 in step 208. 

[0061] Upon receiving a query from a note-taking device 108, presentation server 104 
determines if the user is authorized to access the requested information. The query is not 
processed if the requested information is part of a session which the user is not authorized to 
access. If the user is authorized, then presentation server 104 determines the requested 
20 information and communicates the requested information to the note-taking device that was 
the source of the information request. Accordingly, in step 210, the note-taking device 108 
receives the requested information from presentation server 104. In step 212, the requested 
information received in step 210 is inserted into the notes document in the user-specified 
location. This terminates the processing of the request received in step 202. 

25 [0062] As previously indicated, various different types of recorded information (e.g., slides, 
images, text, audio information, video information) may be requested and inserted in the 
user's notes document. According to an embodiment of the present invention, when audio 
information is inserted in the notes document, a small icon is placed in the notes document 
where the information is inserted. Clicking on the audio icon automatically opens the 

30 inserted audio clip for playback using a preferred player. 

[0063] According to an embodiment of the present invention, the inserted data is actually 
embedded in the notes document—this is substantially different from hyperlinks that provide 
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links to data. As a result, various operations that can ordinarily be performed on a document 
can be performed on the notes document. For example, the document can be copied, 
distributed, etc. In one embodiment, the authorization to access session information is stored 
separately from the document; thereby allowing distribution of selective data without giving 
5 access to the entire session. It should be noted that a user can take notes and include portions 
of recorded information from different presentations in a single document. 

[0064] If it is determined in step 204 that the request received in step 202 cannot be 
processed at that time, then in step 214, the request is stored or cached in the notes document 
for later processing. Multiple unprocessed information requests may be cached in the notes 

10 document. In one embodiment, each cached request is stored as a property associated with 
the notes document. For example, the request may be stored as custom properties of the 
document (e.g., the "CustomDocumentProperties" associated with a Microsoft Word™ 
document). Each cached request may be assigned a unique identifier, e.g., "mrRequest#", 
where # is a unique request identifier. Uniqueness of cached request identifiers in a notes 

15 document is preserved. For each cached request, the data type associated with the request, 
the request time, and other additional information associated with the request may also be 
stored. 

[0065] Information identifying the location in the notes document where the requested 
information is to be inserted may also be cached in the notes document in step 216. In one 

20 embodiment, the location is cached as a bookmark in the notes document. According to an 
embodiment of the present invention, a visible mark or cue is placed in the notes document 
where the information requested by the cached request is to be inserted. The visible mark 
acts as a reminder to the user that information corresponding to an unprocessed request 1 is to 
be inserted in that location. The visible mark may also provide information identifying the 

25 requested information. 

[0066] The information requests that are cached in the notes document are then processed 
at a later time, as indicated in step 218. For example, the cached requests may be 
automatically processed when note-taking device 108 detects that communication with 
presentation server 104 has been established. In one embodiment, note-taking device 108 
30 may be configured to periodically check communications between note-taking device 108 
and presentation server 104 in order to automatically process the cached requests when 
communication is established. 
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[0067] In another embodiment, an attempt to process cached requests may be made by 
note-taking device 108 responsive to various stimuli. For example, such an attempt may be 
made upon receiving a signal generated in response to a user action (e.g., user opens the notes 
document, user explicitly requests a communication check). The signal may also be 
5 automatically generated, e.g., a signal generated by a timer on a periodic basis, a signal 
generated in response to some event related to the notes document. Upon receiving the 
signal, note-taking device 108 may be configured to check communications between the 
note-taking device 108 and the presentation server 104 and to process cached requests if 
communications are possible. 

10 [0068] When the cached requests are processed in step 218, the processing performed in 
steps 206, 208, 210, and 212 is performed for the cached requests. After a request is 
processed, it is removed from the cache. 

[0069] If an error occurs as a result of processing, the request is also removed from the 
cache and an error message inserted at the location where the requested information would 

1 5 have been inserted. In one embodiment, an error may occur when a user tries to request 
information from an unauthorized session, or when the user makes a request for a current 
slide when no presentation is being captured. The errors can be detected in a number of 
ways. In one embodiment, a process error is detected by note-taking device 108 when 
receiving an explicit error signal from presentation server 104 as a result of request 

20 processing. 

[0070] Microsoft Word™ embodiment 

[0071] This section describes an embodiment of the present invention wherein a Microsoft 
Word™ composing application is used to take notes according to the teachings of the present 
25 invention. This embodiment is not intended to limit the scope of the present invention as 
recited in the claims. The teachings of the present invention can also be used with other 
composing applications. 

[0072] In this embodiment, code modules are provided on the note-taking device 108 that 
provide the note-taking functionality of the present invention and run inside any standard 
30 Microsoft Word™ package. These code modules and forms, along with other layout and 

formatting guidelines may be saved and distributed as a Word template file. These modules 
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cause a menu interface to be shown inside Microsoft Word™ that allows users to configure 
requests for recorded information. For example, a user can specify a slide, an audio segment, 
a video clip, etc. to be incorporated into the user's notes during a presentation. The modules 
provide the functionality needed to support user information requests for inserting portions of 
5 recorded information in the user's notes. In one embodiment, the modules that run on note- 
taking device 108 are implemented using Visual Basic for Applications (VBA). 

[0073] As described above, modules implementing the note-taking functionality according 
to the teachings of the present invention may be distributed as a Word template file. Any 
document opened with this template will have the note-taking functionality of the present 

10 invention. For example, when a new document is to be opened for taking notes using 
Microsoft Word™, the user can select the template that provides the functionality of the 
present invention to open the document. As depicted in Fig. 3, a list 300 of templates may be 
displayed to the user and the user can select can select a template 302 (the "mrCapture.dot" 
template) which provides the note-taking functionality according to the teachings of the 

15 present invention. 

[0074] When a Microsoft Word™ document is opened using the "mrCapture.dot" template, 
a new menu bar providing options for specifying information to be inserted into the notes 
document are added to the Microsoft Word™ tool bar. Fig. 4 depicts a section of a Microsoft 
Word™ tool bar showing a menu bar 402 comprising user-selectable options for inserting 

20 portions of recorded information into a notes document according to an embodiment of the 
present invention. In the embodiment depicted in Fig. 4, menu bar 402 comprises several 
buttons and toggles: button ("Insert") 404, button ("Calendar 1 ') 406, "Slide" toggle 408, 
"Audio" toggle 410, and "Video" button 412. The user can select "Slide" toggle 408 to 
indicate that the presently displayed slide should be inserted in the user's notes document. 

25 The position of the cursor within the notes document at the time when "Slides" toggle 408 is 
clicked indicates the position in the document where the slide is to be inserted. Likewise, the 
user can set "Audio" toggle 410 to indicate that an audio segment is to be inserted. Users can 
configure the length of audio clip to be inserted using pull-down menu 414 associated with 
"Audio" toggle 410. When the user clicks the "Insert" button 404, the specified information 

30 will be inserted in the user's notes document at the position of the cursor. It should be 

apparent that buttons and/or toggles for other types of information may be provided in other 
embodiments of the present invention. 
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[0075] As previously described, a user can be authorized to access information from 
multiple sessions and can incorporate portions of recorded information from these sessions 
into the user's notes. Selecting ("Calendar") button 406 in Fig. 4 brings up a dialog box 500 
as depicted in Fig. 5 that displays information regarding the various other sessions from 
5 which the user is authorized to access information according to an embodiment of the present 
invention. A calendar interface is displayed in Fig. 5. The user can select a range of dates 
from the calendar interface. As shown in Fig. 5, the user has selected a range of days 
including October 6, 2002 to October 11, 2002. A list of sessions 502 that a user is 
authorized to access within the selected range of days is displayed. Summary information for 
10 each session is displayed including the starting time of the recording included in the session, 
the duration of the session information, and the number of slides included in the session 
information. 

[0076] While attending and taking notes for a presentation, the user can specify that a 
portion of recorded information from another presentation, which is stored as another session 
15 to which the user is authorized, be inserted into the user's notes. The user can specify the 
particular session from which information is to be inserted. For example, in Fig. 5, the user 
can select one of the sessions displayed in list 502. The user can then specify the portion of 
information to be inserted. For example, the user can specify a slide number to be inserted. 

[0077] If the recorded information for user-selected session is accessible to the note-taking 
20 device used by the user, then the recorded information from that session may be displayed to 
the user on the note-taking device 108 and the user can then pick and chose the information 
to be inserted in the user's notes. The recorded information for the selected session may be 
stored by presentation server 104 and accessible to note-taking device 108 via server 104. In 
some embodiments, the recorded information for sessions which are authorized for the user 
25 may be downloaded to the user's note-taking device and cached in the note-taking device's 
memory. According to an embodiment of the present invention, the download may be 
performed when the user logs in and get authorized for accessing information recorded for 
the present presentation. 

[0078] Fig. 6 displays a user interface that allows a user to browse through previously 
30 recorded session information when the information is accessible to the user's note-taking 
device according to an embodiment of the present invention. Once the user has selected a 
session using the user interface depicted in Fig. 5, the user can select "Slide" tab 504 to bring 
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up the user interface displayed in Fig. 6 which displays slides recorded for the selected 
session. As shown, recorded slides are displayed in window 602. The user can browse 
through the slides using slider 604 or step through them one at a time using previous and next 
buttons 606. Several control buttons 608 are provided for inserting a slide or audio segment 
5 into the user f s notes. Controls 61 0 are also provided for playing back the recorded 
information stored for the session. A slide counter 612 is also displayed. 

[0079] If information for a selected slide is accessible to the note-taking device, either via 
presentation server 104 or cached on the note-taking device itself, the request for the selected 
slide is immediately processed and the requested data is inserted into the user's notes 
10 document. If the request information is not accessible, then the information request is cached 
in the notes document, and processed later when the requested information becomes 
accessible, as described above. 

[0080] As described above, the user interfaces depicted in Figs. 5 and 6, and described 
above allow the user to specify information to be inserted into the user notes from previously 

15 recorded sessions which the user is authorized to access. The user interfaces can also be used 
to view and select information previously displayed during the present presentation that the 
user is attending and which is stored as part of the current session. For example, using the 
interface depicted in Fig. 6, the user can step back through slides that have already been 
displayed in the presentation the user is attending and select a slide for insertion in the user's 

20 notes. As new slides are displayed during the presentation, slide counter 612 is updated to 
include the new slides. 

[0081] Fig. 7 A depicts a Microsoft Word™ interface 700 that has been configured to 
provide options for taking notes according to an embodiment of the present invention. 
Interface 700 displays a section 702 of a user's notes document. As shown, a visible marker 

25 704 is displayed in the notes document corresponding to an information request that has not 
yet been processed and is accordingly stored or cached in the notes document. Visible 
marker 704 is placed in a location where the information requested by the cached information 
request is to be inserted. Visible marker 704 provides a context of the requested information 
such as "[mrRequestl : To be inserted slide4 from 13:09:23 10/1/0/02]". The user may have 

30 requested the information to be inserted dialog window 706 (which is similar to the user 
interface depicted in Fig. 6 and described above). The information request is stored in the 
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notes document until the request can be processed, i.e., the information requested by the 
request can be accessed and inserted into the notes document. 

[0082] Fig. 7B depicts the notes document of Fig. 7A wherein the cached information 
request has been processed according to an embodiment of the present invention. As shown, 
5 a slide 708 corresponding to the cached information request has been inserted into the notes 
document at the specified location. Visible marker 704 has accordingly been automatically 
removed and replaced by the requested slide 708. Audio information associated with the 
inserted slide has also been inserted into the notes document as indicated by icon 710. The 
user can playback the audio segment by clicking or selecting icon 710. 

10 [0083] As described above, the code modules that provide the functionality needed to 

support user information requests for inserting portions of recorded information in the user's 
notes may be implemented in Visual Basic for Applications (VBA) that is supported as part 
of the standard Microsoft Office Suite™. Several code modules may be provided. For 
example, a "user request module" may be provided that provides the functionality needed to 

15 support user information requests such as inserting slides, audio segments, and so on. The 
"user request module" may interact with different types of objects including a Word 
document object, a user request object, and a presentation or session object. 

[0084] According to an embodiment of the present invention, the user request object is a 
class object that handles creation of a user information request (e.g., a request created using 

20 the interfaces depicted in Figs. 5 and 6), processing the user information request to obtain the 
requested information and inserting the information in the notes document, caching the 
request when it cannot be processed, and loading in cached requests during module startup. 
The session object comprises information about presentations (or sessions) and is configured 
for formulating a query for the information request, communicating the query to the 

25 presentation server 104, receiving requested information from the presentation server 104, 
managing cached sessions on the note-taking device, and processing of login keys. The 
Word object is part of the Microsoft Office Object Model that is provided for manipulation of 
document elements. 

[0085] Fig. 8 is a simplified block diagram of a note-taking device 108 that may be used to 
30 perform processing according to an embodiment of the present invention. As shown in Fig. 
8, note-taking device 108 includes at least one processor 802, which communicates with a 
number of peripheral devices via a bus subsystem 804. These peripheral devices may include 
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a storage subsystem 806, comprising a memory subsystem 808 and a file storage subsystem 
810, user interface input devices 812, user interface output devices 814, and a network 
interface subsystem 816. The input and output devices allow user interaction with data 
processing system 802. 

5 [0086] Network interface subsystem 816 provides an interface to other computer systems 
such as presentation server 104 and authorization device 116. The networks may include the 
Internet, a local area network (LAN), a wide area network (WAN), a wireless network, an 
intranet, a private network, a public network, a switched network, or any other suitable 
communication network. Network interface subsystem 816 serves as an interface for 

10 receiving data from other sources and for transmitting data to other sources from note-taking 
device 108. For example, note-taking device 108 may send information requests to 
presentation server 104 and receive the requested information from presentation server 104 
via network interface subsystem 816. Embodiments of network interface subsystem 816 
include an Ethernet card, a modem (telephone, satellite, cable, ISDN, etc.), (asynchronous) 

15 digital subscriber line (DSL) units, and the like. 

[0087] User interface input devices 812 may include a keyboard, pointing devices such as a 
mouse, trackball, touchpad, or graphics tablet, a scanner, a barcode scanner, a touchscreen 
incorporated into the display, audio input devices such as voice recognition systems, 
microphones, and other types of input devices. In general, use of the term "input device" is 
20 intended to include all possible types of devices and ways to input information to note-taking 
device 108. A user may use user input devices 812 to take notes. 

[0088] User interface output devices 814 may include a display subsystem, a printer, a fax 
machine, or non- visual displays such as audio output devices. The display subsystem may be 
a cathode ray tube (CRT), a flat-panel device such as a liquid crystal display (LCD), or a 
25 projection device. In general, use of the term "output device" is intended to include all 
possible types of devices and ways to output information from note-taking device 108. 

[0089] Storage subsystem 806 may be configured to store the basic programming and data 
constructs and modules that provide the functionality of the present invention. For example, 
software modules such as authorization module 118 may be stored in storage subsystem 806. 
30 These software modules may be executed by processor(s) 802. Storage subsystem 806 may 
also provide a repository for storing data used in accordance with the present invention. For 
example, the notes documents, cached sessions information, etc. may be stored in storage 
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subsystem 806. Storage subsystem 806 may comprise memory subsystem 808 and file/disk 
storage subsystem 810. 

[0090] Memory subsystem 808 may include a number of memories including a main 
random access memory (RAM) 818 for storage of instructions and data during program 
5 execution and a read only memory (ROM) 820 in which fixed instructions are stored. File 
storage subsystem 810 provides persistent (non- volatile) storage for program and data files, 
and may include a hard disk drive, a floppy disk drive along with associated removable 
media, a Compact Disk Read Only Memory (CD-ROM) drive, an optical drive, removable 
media cartridges, and other like storage media. 

10 [0091] Bus subsystem 804 provides a mechanism for letting the various components and 
subsystems of note-taking device 108 communicate with each other as intended. Although 
bus subsystem 804 is shown schematically as a single bus, alternative embodiments of the 
bus subsystem may utilize multiple busses. 

[0092] As previously stated, note-taking device 108 can be of varying types including a 
1 5 personal computer, a portable computer, a laptop, a kiosk, a PDA, a tablet PC, and the like. 
The description of note-taking device 108 depicted in Fig. 8 is intended only as a specific 
example. Many other configurations having more or fewer components than the system 
depicted in Fig. 8 are possible. 

[0093] Although specific embodiments of the invention have been described, various 
20 modifications, alterations, alternative constructions, and equivalents are also encompassed 
within the scope of the invention. The described invention is not restricted to operation 
within certain specific data processing environments, but is free to operate within a plurality 
of data processing environments. Additionally, although the present invention has been 
described using a particular series of transactions and steps, it should be apparent to those 
25 skilled in the art that the scope of the present invention is not limited to the described series 
of transactions and steps. 

[0094] Further, while the present invention has been described using a particular 
combination of hardware and software, it should be recognized that other combinations of 
hardware and software are also within the scope of the present invention. The present 
30 invention may be implemented only in hardware, or only in software, or using combinations 
thereof. 
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[0095] The specification and drawings are, accordingly, to be regarded in an illustrative 
rather than a restrictive sense. It will, however, be evident that additions, subtractions, 
deletions, and other modifications and changes may be made thereunto without departing 
from the broader spirit and scope of the invention as set forth in the claims. 
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